1. Field of the Invention
The present invention relates to an image processing device, an imaging apparatus, an image processing method and a program and, more particularly, an image processing device for performing a correction process of a pixel value of a synthesized image generated using a plurality of images, an imaging apparatus, an image processing method, and a program.
2. Description of the Related Art
A solid-state imaging element, such as a CCD image sensor or a Complementary Metal Oxide Semiconductor (CMOS) image sensor, used in a video camera, digital camera or the like, accumulates charges according to an incident light amount and performs photoelectric conversion for outputting an electrical signal corresponding to the accumulated charges. However, since there is an upper limit in charge accumulation amount of a photoelectric conversion element, if a certain amount of light or more is received, the charge accumulation amount reaches a saturation level. Thus, a so-called whitening-out fixed at a saturated luminance level occurs in a subject region having a certain brightness or greater.
In order to prevent such a phenomenon, a process of controlling a charge accumulation period of the photoelectric conversion element according to a variation in external light or the like so as to adjust an exposure time such that sensitivity is controlled to an optimal value is performed. For example, a shutter is released with respect to a bright subject at a high speed so as to shorten an exposure time and the charge accumulation period of the photoelectric conversion element is shortened so as to output the electric signal before the charge accumulation amount reaches the saturation level. By such a process, it is possible to output an image capable of accurately ensuring reproduction of the grayscale according to the subject.
However, in photographing a subject in which a bright place and a dark place are mixed, if the shutter is released at a high speed, a sufficient exposure time may not be taken for the dark place. Thus, an S/N ratio deteriorates and image quality deteriorates. In an image obtained by photographing the subject in which the bright place and the dark place are mixed, in order to accurately reproduce the luminance level of a dark portion, a process of realizing a high S/N ratio by a long exposure time in pixels in which the amount of incident light on an image sensor is small and avoiding saturation in pixels in which the amount of incident light is large is necessary.
As a method of realizing such a process, a method of using a plurality of images having different exposure times is known. That is, a method of using a long-time exposure image in a dark image region and using a short-time exposure image in a bright image region in which whitening-out seems to occur in the long-time exposure image so as to determine optimal pixel levels may be used. By synthesizing a plurality of different exposure images, it is possible to obtain an image with a wide dynamic range, in which whitening-out does not occur.
For example, Japanese Unexamined Patent Application Publication No. 2008-99158 or Japanese Unexamined Patent Application Publication No. 2008-227697 discloses a configuration in which an image with a wide dynamic range is obtained by synthesizing a plurality of images having different exposure amounts. This process will be described with reference to FIG. 1. An imaging device, for example, outputs image data of two different exposure times within a video rate (30 to 60 fps) in moving-image photographing. In addition, image data of two different exposure times is generated and output in still-image photographing. FIG. 1 is a diagram illustrating the characteristics of images having two different exposure times (long-time exposure image and short-time exposure image) generated by the imaging device. A horizontal axis denotes a time t and a vertical axis denotes the charge accumulation amount e in a light sensing photodiode (PD) configuring a photoelectric conversion element corresponding to one pixel of a solid-state imaging element.
For example, in a bright subject in which the light sensing amount of the light sensing photodiode (PD) is large, as shown in a high luminance region 11 of FIG. 1, the charge accumulation amount is rapidly increased with the elapse of time. In contrast, in a dark subject in which the light sensing amount of the light sensing photodiode (PD) is small, as shown in a low luminance region 12 of FIG. 1, the charge accumulation amount is slowly increased with the elapse of time.
A time t0 to t3 corresponds to an exposure time TL for acquiring the long-time exposure image. In a line indicated as the low luminance region 12 of the long-time exposure time TL, the charge accumulation amount does not reach a saturation level at a time t3 (non-saturation point Py), and accurate grayscale expression is obtained by the grayscale level of the pixel determined using an electric signal obtained by the charge accumulation amount Sa.
However, in a line indicated as the high luminance region 11, the charge accumulation amount reaches the saturation level (saturation point Px) before reaching the time t3. Accordingly, in the high luminance region 11, only a pixel value corresponding to an electric signal of the saturation level is obtained from the long-time exposure image and, as a result, whitening-out occurs in the pixel.
In the high luminance region 11, at a time before reaching the time t3, for example, at a time t1 (charge sweep-out start point P1) shown in the drawing, the charges accumulated in the light sensing photodiode (PD) are swept out. The charges are swept out not to all charges accumulated in the light sensing photodiode (PD) but to an intermediate voltage retention level controlled in the photodiode PD. After the charge sweep-out process, short-time exposure is executed again during an exposure time TS (t2 to t3). That is, the short-time exposure is performed from the short-time exposure start time P2 to the short-time exposure end time P3 shown in the drawing. By the short-time exposure, the charge accumulation amount Sb is obtained, and the grayscale level of the pixel is determined based on the electric signal obtained based on the charge accumulation amount Sb.
In addition, when the pixel value is determined based on the electric signal based on the charge accumulation amount Sa obtained by the long-time exposure in the low luminance region 12 and the electric signal based on the charge accumulation amount Sb obtained by the short-time exposure in the high luminance region 11, an estimated charge accumulation amount when the same-time exposure is performed or an electric signal output value corresponding to the estimated charge accumulation amount is calculated and a pixel value level is determined based on the calculated result.
By combining the short-time exposure image and the long-time exposure image, it is possible to obtain an image with a wide dynamic range, in which whitening-out does not occur.
However, the plurality of images having different exposure amounts becomes images photographed at different timings. Accordingly, when the subject is moved during photographing, a deviation in images occurs when the images are synthesized. As a result, a pseudo color is generated in an image portion of a moving subject region and thus image quality deteriorates. In addition, even when the subject is not moved, the same image quality deterioration may occur if the luminance of the subject is changed.
Japanese Unexamined Patent Application Publication No. 2008-227697 discloses a pixel value correction method for solving such a problem. As described above, in the “image with the wide dynamic range” generated by the synthesized image, the pixel value becomes an erroneous value in each position of a pixel position in which the luminance of the subject is changed from high luminance to low luminance and a pixel position in which the luminance of the subject is changed from low luminance to high luminance and thus grayscale failure of the pixel occurs or a pseudo color is generated in the image.
In Japanese Unexamined Patent Application Publication No. 2008-227697, correction is executed with respect to such an image so as to suppress the grayscale failure of the pixel or generation of the pseudo color. The correction method will now be described. When the pixel value of an original image is Dv and the pixel value of the same position in the image obtained by performing a blurring process with respect to the original image is Mv, coefficients a and b (a+b=1) are applied and the pixel value [Rv] of the same position of the output pixel determined as the pseudo color is calculated as follows.[Rv]=a×[Dv]+b×[Mv]
Since the blurring process has a purpose of applying, for example, a general low-pass filter (LPF) and blurring the image in regard to the applied filter size (the number of taps), two-dimensional expansion of the imaged pixel as a luminance variation is considered.
From the viewpoint of a high possibility that the pseudo color generated in the vicinity of the moving subject has a high frequency component and is not important, the LPF is used. However, if the expansion of the pixel to be corrected is large, the assumption is not satisfied.
In order to correct a place in which the expansion of the pixel to be corrected is large, the number of taps of the filter is set to be large. Thus, since hardware implementation is difficult, information amount is reduced by an image reduction process and the reduced image is enlarged again so as to obtain the same effect as the blurring process. However, according to a photographing environment, a pseudo color having high chroma is output with respect to the moving subject in a large area and the color may be conspicuous even in the blurred image.
In order to avoid this problem, a chroma reduction process is performed with respect to the image to which the LPF is applied. However, in the chroma reduction process, in particular, the correction of the large pseudo color is not sufficient. In addition, in the pixel in which it is erroneously determined that the pseudo color is generated in the moving subject, chroma reduction may be conspicuous.